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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



Applicant: 
Serial No..- 
Filing Date: 



Title: 



James R Wright Examiner: Shahid Al Alam 

09/757,849 Art Unit: 2172 

January 9, 2001 

System for Searching Collections of Linked Objects 



Commissioner for Patents 
P.O. Box 1450 
Alexandria, VA 22313-1450 

CERTIFICATE OF MAILING 
I hereby certify that this correspondence is being sent to the United States Patofit and Trademark Office by facsimile 
transmission to 571-273-8300 on February 20, 2006. A /' /t 



In response to the Office communication of January 20, 2006, regarding the above 
numbered Application, and per the helpful instructions of the Examiner during our conversations 
in the interim, please find attached a new copy of the brief first submitted on April 30, 2004 with a 
revised Summary of the Claimed Subject Matter, section headings and appendices. However, it is 
noted that this appeal brief (and the rejected brief filed June 1, 2005 and referred to in the January 
20 Office communication) is being filed pro se. The Applicant thus respectfully submits that the 
ground of rejection offered for the brief in the January 20 Office communication therefore was, 
and is, inapplicable. 

37 CFR § 41.37(c)(1) states that "a brief filed by an appellant who is not represented by a 
registered practitioner need only substantially comply with paragraphs (c)(l)(i) through (c)(l)(iv) 
and (c)(l)(vii) through (c)(l)(x) of this section",. The express basis of both (a) the checked 
rejection (#4) of the January 20 Office communication, and (b) the requirement that the brief 
contain a Summary of the Claimed Subject Matter, is 37 CFR § 41.37(c)(l)(v), but by the terms of 
§ 41.37(c)(1), § 41.37(c)(l)(v) is inapplicable to this brief because it is being filed pro se. The 
Applicant therefore respectfully submits that the rejection of the appeal brief for alleged failures in 
the Summary of the Claimed Subject Matter was unnecessary in accordance with the applicable 
regulation. The Applicant further respectfully submits that the secondary ground for rejection set 
forth in the "Other" section of the January 20 Office communication, regarding the section's title, 
is not necessary either, pursuant, e.g., to MPEP § 1205,03, which states that "The examiner should 
not require a corrected brief for minor non-compliance in an appeal brief (e.g., the brief has a 
minor error in the title of a section heading)" 



Sir: 
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Finally, the Applicant respectfully directs the Office's attention to the fact that this Appeal 
and Brief have now been pending for over 21 months without substantive response. Therefore, in 
order to expedite the resolution of the issues on appeal, the Applicant respectfully requests an 
immediate transfer to the Board of Patent Appeals for this Appeal and Brief If there is an 
additional appeal brief fee beyond the fee already charged in respect of the original submission, 
please charge it using the attached credit card charge authorization form, however, in light of the 
fact that the rejections causing resubmission were not in conformance with the applicable 
regulations, and the appeal brief fee for this appeal has already been paid, the Applicant 
respectfully submits that no new charge for this resubmission is warranted or should be required. 



James E. Wright 
CommSoft Corporation 
P,0, Box 712280 
Los Angeles, C A 90071 
(818) 632-7135 




2 



PACE 3/50 * RCVD AT 2/20/2006 10:25:04 PM [Eastern Standard Time] * SVR:USPTO-EFXRF-6/26 * DNIS: 2738300 * CSID: Sidley * DURATION (mm-ss):79-26 



02/20/2006 07:27:53 PM 



Sidley 



Sidley 



Page 5 




% S 7006 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 



Applicant: 
Serial No.: 
Filing Date: 



Title: 



James E. Wright Examiner: Shahid Al AJam 

09/757,849 Ait Unit: 2172 

January 9, 2001 

System for Searching Collections of Linked Objects 



Commissioner for Patents 
P.O. Box 1450 
Alexandria, VA 22313-1450 




f hereby certify that this correspondence is being sent to the United States Fatent and Trademark Office by facitmilie 
transmission to 571-273-8300 on February 20, 2006. / I jc 



APPELLANT'S BRIEF UNDER 37 CFR 3 1,192 

Real Party in Interest 

The real party in interest in the present application is CommSoft Corporation, by virtue of 
an assignment by the inventor, James Wright, recorded at Reel 01 1850, Frame 0153. 

Related Appeals and Interferences 

There are no related appeals or interferences. 

Status of Claims 

Claims 1 -17, 1 9, and 20 are pending in the application. Claim 1 8 was cancelled by the 
Office Action Response filed February 26, 2003. 

Status of Amendments 

There are no unentered amendments. 



Sir: 
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Summary of the Claimed Subject Matter 

The present invention includes methods of searching collections of linked objects and 
displaying the results. The invention is of particular utility when the links between objects (e.g., 
legal citations or bibliographic references) themselves tend to convey useful information about 
the objects or their relevance to the search. 

According to the inventive methods of independent Claims 1 and 2, once a search group 
is acquired and a set of links therefrom determined (as discussed, e.g., in paragraphs 1 - 3 of the 
Detailed Description, on numbered page 2 of the July 1 1 , 2002 published application, and as 
depicted, e.g., in Figure 2), the links from the target objects are used to determine at least one 
display attribute of the search set when it is displayed to the user. For example, links may be 
displayed as arrows or other connectors on a graph (as required by independent Claim 1 6 and 
shown, e.g., in Figure 2), or otherwise, or color, shape, size, position, highlighting, graphical 
flags, and/or labeling text may be used to convey information about the links (as discussed;, e.g., 
in paragraph 4 of the Detailed Description, on page 2 of the published application, and shown, 
e.g., in Figure 5)* 

The objects and links displayed may furthermore be layered as additionally required by 
independent Claim 2 (as shown, e.g. P in Figures 2, 3, and 4, and discussed, e.g. , in paragraphs 4 - 
8 of the Detailed Description, on pages 2 and 3 of the published application). The objects may 
also be annotated with user notes or other information as required by independent Claim 16 (as 
discussed, e.g., in paragraphs 4 and 6 of the Detailed Description, on pages 2 and 3 of the 
published application). 

Issues 

There are two issues presented: 

1. Whether claims 1, 16, 17, 19, and 20 are anticipated by Shah, et aL, "Infohamess: 
Managing Distributed, Heterogeneous Information" IEEE Internet Computing, Nov/Dec 1999 
(hereinafter, "Shah"). 

2. Whether claims 2-15 are obvious over Shah in view of U.S. Patent No, 5 7 983 ? 267 
to Shklar, et al (hereinafter, "Shklar")- 

2 
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Grouping of Claimg 

The claims do not stand or fall together, for the reasons set forth in the following section. 

Argument 

The Prior Art 

Shah describes a metadatabase system for indexing heterogeneous groups of documents 
and images. The system allows searching of metadata to locate documents, and allows a user to 
create associations between objects. For example, a user may define a group of documents that 
all relate to the same subject. Shah calls the creation of such associations is called "annotation." 

Shklar describes another system for representing data of heterogeneous types. The 
system tries to provide a uniform presentation format for the heterogeneous data by analyzing its 
internal organization {e.g., into paragraphs, sections, articles* chapters, or frames), and displaying 
selected portions of the data. 

Claim 1 

Independent claim 1 recites a method of searching a collection of linked objects and 
displaying the results. According to the method, a search group of heterogeneously typed objects 
is acquired (for example, by keyword searching or other known techniques). The objects are 
characterized in that at least one of the searched objects comprises a link to another object (for 
example, a bibliographic citation from one paper to another). For at least a portion of the objects 
in the acquired search group, the internal links are "followed" to determine their target objects. 
This determination includes determining whether the target objects are themselves inside or 
outside of the search group. Finally, at least one member of the acquired search group is 
displayed, where some aspect of the display (a "display attribute") is determined by the set of 
target objects. (For example, an object may be color-coded to indicate that it contains links to 
other objects within the search group). 

Claim 1 currently stands rejected as anticipated by Shah. However, it contains two 
features not found in or suggested by Shah: determination of link targets and using the set of 
linked targets to determine a display attribute. While Shah allows a user to view a document 
retrieved from a search, neither user nor search engine necessarily follows any embedded links in 
that document to determine their targets. The mere loading of an object containing a link as 

3 
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suggested by Shah clearly does not determine the identity of the link target as required by the 
claim. A single target may have multiple incoming links, for example through different domains 
by reference to the examiner's HTML reference, and likewise, links may be either relative or 
absolute, or dynamic, such that the same apparent link "address'* from two different objects or 
even firom the same object at different times might yield two different link targets. 

Further, and critically, there is absolutely no suggestion that either search engine or user 
should determine whether link targets are inside or outside of the originally acquired search 
group, as recited by claim 1. In fact, failing to probe the target objects during the search to 
determine their identities, there is no way to even determine whether a link target would be 
inside the original search result group, since the link address from one object (e.g., an HTML 
page) to another might not be the same "address" followed to reach the second object as it was 
located in the original search. 

In addition, there is no disclosure in Shah of using the set of targets linked to by objects 
in a search group to determine display attributes of the group. The Examiner has suggested that 
this feature is satisfied simply by displaying an HTML document with underlined links. 
However, again, in this case, the display attribute (underlining) is not determined by the targets 
of the links, which are not loaded or searched by the browser, and in fact may not even exist (e.g. 
dead or broken links), but solely by the original HTML page and the browser settings. The 
undetermined link targets therefore cannot possibly be the basis for changing the display 
attributes of the group or in this example the originally displayed object (the HTML page). 

Since claim 1 contains two features not found in Shah, it cannot be anticipated under 35 
U.S,C. § 102 by that reference. In addition, it is not obvious over Shah, because these two 
features are not suggested or in any way obvious modifications of that reference. 

Claim 2 

Independent claim 2 includes all of the features of claim 1, and also recites that displayed 
representation of search objects are arranged in layers, which may be independently hidden or 
shown by a user. This claim currently stands rejected as obvious over Shah in view of Shklar. 
The arguments set forth above in connection with claim 1 and Shah apply equally to claim 2. 
Nothing in Shah renders the features of determination of link targets and using the set of linked 

4 
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targets to determine a display attribute as obvious. These deficiencies are not remedied by 
Shklar, which is relied upon primarily to teach the use of layers. 

To the extent that Shklar discusses links at all, it describes internal links within a single 
document, linking one section of the document to another. In contras t claim 2, like claim 1, 
recites that an object comprises a link to another object - an "external" link. Thus, the links in 
Shklar are not even the same type as those recited in claim 2. Further, nothing in Shklar suggests 
determining link targets as recited in claim 2. Not only is there no disclosure of determining 
whether link targets are inside or outside of the originally acquired search group, such a 
determination would not even make sense in the context of internal links (since by definition, 
they point to an object in the search group - the same object from which they originate). 
Similarly, the set of target objects is not used to determine a display attribute, because the set of 
target objects is always the same for a document with only internal links - it consists solely of 
the originating object. 

Since neither Shall nor Shklar suggests determining whether linked objects are inside or 
outside the search group* or using the set of linked objects to determine a display attribute, they 
cannot render claim 1 or claim 2 obvious under 35 U.S.C. § 103. 

Claims 3-15 

Claims 3-15 depend (directly or indirectly) in the alternative from either claim 1 or claim 
2, and thus cannot be considered obvious over the combination of Shah and Shklar for the 
reasons set forth above. However, certain of these claims also recite additional features not 
found in either reference, and thus must independently be considered nonobvious. 

Claim 4 

Claim 4 recites that representations of a plurality of objects from a search set are 
displayed on a graph. The specification defines a graph at page 4, lines 1 1 -13, as "a two- 
dimensional or three-dimetisional visual representation of linked objects, where a link is 
displayed as a connector." An example of representations of objects displayed on a graph may 
be found in Figure 2 of the present application. Nothing in either Shah or Shklar suggests 
showing representations of linked objects on a graph. The Examiner has cited page 22, right 
column, lines 3-23 of Shah for this proposition, but that reference teaches only display of a single 

5 
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object itself, rather than a display of representations of multiple related objects on a graph. No 
connectors are shown or described, and only the single object is shown. 

Claim 5 

Claim 5 recites that representations of a plurality of objects are displayed, and that at least 
one link between objects is depicted by a connector between these representations. As discussed 
above, neither Shah nor Shklar teaches or suggests displaying a plurality of object 
representations and a connector between representations. 

Claim 6 

Claim 6 depends from claim 5, and further adds that the connector has a display attribute 
determined by (i) the source object type, (ii) the target object type, or (iii) the link type. Shah 
and Shklar do not even show connectors, and certainly do not display those connectors 
differently depending on type. Further, neither reference makes any suggestion that links may 
have different types, 

Claim S 

Claim 8 recites that deterannfng link targets includes not only determining the set of all 
objects linked to by the original search group, it further includes recursively determining the set 
of all objects linked to by the expanded set of objects consisting of the union of the search group 
and the set of objects linked to by the search group. Shah and Shklar do not determine the set of 
objects linked to by a search group, and certainly do not determine "second-order" links by 
following a chain of outward links. 

Claims 16, 17, 19, and 20 

Independent claim 16 recites a search method that includes annotating the results of a 
search. That is, a user may attach notes to individual objects returned by the search, and these 
notes may be selectively displayed when viewing the search results. The search results are 
displayed on a graph. This claim (and its dependent claims 17, 19, and 20) currently stands 
rejected as anticipated by Shah. 

As discussed above, Shah does not teach or suggest the display of representations of 
search objects on a graph, and for that reason alone, cannot anticipate claims 16, 17, 19, and 20. 

6 
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In addition, while Shall does use the term "annotate " it does not do so in the same sense as the 
present application, and cannot be considered to describe this feature of claim 16. In Shah, 
"annotation" is not the attaching of a note to a specific object, but is the classification of an 
object into a group by establishing an "annotation relationship" between objects. Such a 
relationship cannot anticipate the attachment of a note to a single object. Further, Shah does not 
describe the selective display of annotations, as also recited by claim 1 6. 



In addi tion, dependent claim 20 adds that representations of objects arc displayed with 
connectors representing links between the objects, and that these links may be annotated. As 
discussed above. Shah does not teach the display of connectors to represent links between 
objects, and certainly does not teach the annotation of such links. Thus, claim 20 is also not 
anticipated for this independent reason. 



For the reasons set forth above, claims 1, 16, 17, 19, and 20 are not anticipated by Shah, 
and claims 2-15 are not obvious over Shah in view of Shklar. Applicant therefore respecfully 
requests that the Board remove all rejections and allow the present claims to pass to issuance. 

If there is an additional appeal brief fee beyond the fee already charged in respect of the 
original brief, please charge it using the attached credit card charge authorization form. 



James E. Wright 
CommSofl Corporation 
P.O. Box 712280 
Los Angeles, CA 90071 
(818) 632-7135 



Claim 20 



Conclusion 
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CLAIMS APPENDIX 

1 . (Original) A method of searching a collection of linked objects and displaying the results, 
comprising: 

acquiring a search group of heterogeneously typed objects, wherein at least one of the 

objects comprises a link to another object; 
determining for at least a portion of the objects in the search group a set of targets of links 

from the objects, including determining whether the link targets are inside the 

search group; and 

displaying a representation of at least one searched object, the representation having at 
least one display attribute determined by the set of link targets. 

2. (Original) A method of searching a collection of linked objects and displaying the results, 
comprising: 

acquiring a search group of objects, wherein at least one of the objects comprises a link to 
another object; 

determining for at least a portion of the objects in the search group a set of targets of links 
from the objects, including determining whether the link targets are inside the 
search group; and 

displaying a representation of at least one searched obj ect, the representation having at 
least one display attribute determined by the set of link targets, 

wherein displayed representations are arranged into a plurality of display layers, and 
wherein the display layers can be independently hidden or displayed. 

3. (Original) The method of claim 1 or 2, wherein the display attribute is selected from the 
group consisting of color, shape, size, position, highlighting, graphical flags, and labeling 
text. 

4. (Original) The method of claim 1 or 2. wherein representations of a plurality of objects 
are displayed on a graph. 

8 
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5. (Original) The method of claim 1 or 2, wherein representations of a plurality of objects 
are displayed, and wherein at least one link between objects is depicted by a connector 
between the representations. 

6. (Original) The method of claim 5, wherein a display attribute of the connector is 
determined by a property selected from the group consisting of the type of the linking 
object, the type of the link target, and the type of the link. 

7. (Original) The method of claim 1 or 2, wherein a display attribute of the representation is 
determined by object metadata. 

8. (Original) The method of claim 1 or 2, wherein determining link targets includes 
recursively determining targets of links of an expanded set of objects comprising the 
original search group and the objects linked to by the search group. 

9. (Original) The method of claim 8, wherein the recursion level is in the range of 1-10. 

10. (Original) The method of claim 1 or 2, wherein the search objects comprise documents 
selected from the group consisting of legal opinions, legal treatises, statutes, briefs, and 
law review articles. 

1 1 . (Original) The method of claim 1 or 2, wherein the search objects comprise scientific or 
medical writings, and wherein the links comprise citations to other scientific or medical 
writings. 

12. (Original) The method of claim 1 or 2, wherein the search objects comprise patents and 
patent applications and the links comprise references to related patents and patent 
applications. 

I3 a (Original) The method of claim 1 or 2, further comprising annotating at least one of the 
search objects. 
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14. (Original) The method of claim 1 or 2, wherein at least a portion of the searched objects 
and link targets are classified into a plurality of groups, further comprising setting a 
display attribute for all members of a group. 

1 5 . (Original) The method of claim 1 or 2, wherein displayed representations are sorted on at 
least one axis according to a property of the objects represented. 

16. (Previously presented) A method of searching a collection of objects and displaying the 
results, comprising: 

acquiring a first search group of objects; 

displaying a representation of at least a portion of the first search group of objects on a 
graph; and 

annotating one or more members of the first search group of objects, wherein annotations 
may be selectively displayed with the representation of the annotated objects. 

17. (Original) The method of claim 16, further comprising: 
acquiring a second search group of objects; and 

displaying a representation of at least a portion of the second search group of objects, 
wherein displaying the representation of annotated objects that are members of 
both the first search group and the second search group includes selectively 
displaying annotations of the objects. 

18. (Cancelled) 

19. (Original) The method of claim 16, wherein the objects include links to other objects, and 
wherein at least a portion of the links are displayed as connectors between representations 
of the objects. 

20. (Original) The method of claim 19, further comprising annotating one or more links. 
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EVIDENCE APPENDIX 

Each of the following items was entered in the record by the Examiner's Office Action of 
November 1 T 2002: 

Item 1 * Shall, et al., "Infoharness: Managing Distributed, Heterogeneous Information", IEEE 
Internet Computing* Nov/Dee 1999* 

Item 2: U.S. Patent No. 5,983,267 to Shklar, et al 
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Using metadata extraction 
methods, InfoHarness provides 
integrated, rapid access 
to huge amounts of 
heterogeneous information, 
regardless of type, 
representation, location, 
and medium = 




KsHmj Shah and amit Smith 

University of Georgia, Large-Scale Distributed Information Systems Lab 



jj oday, important information is scattered in so many places, formats* 
and media, that getting die right information at the right time and 
place is an extremely difficult task. Developing a single software 
product, for example, includes the creation of documenrs ranging from 
the requirements specification and project schedules ro marketing pre- 
sentations, multimedia tutorials, and more. Each document may be cre- 
ated by a different person using a different tool, and each may be scored 
in a different place. 

InfoHarness is an information integration system, platform, and tool 
set that addresses these problems, managing huge amounts of heteroge- 
neous information in a distributed environment. Through a powerful, 
consistent user interface* InfoHarness provides rapid search of and access 
to uuorniation assets including documents and parts of documents, mail 
messages, images, code files, video clips, Web pages with URLs, InfoHar- 
ness queries* and views of relational tables. The system makes all these aiti- 
faccs available without relocating, restructuring, or reformatting the data. 

Instead, InfoHarness associates each original artifact widi an extensi- 
ble set of metadata — for example, the artifact's type, locanon, access rights, 
owner, and creation date. Using the metadata, the system rapidly, and 
largely automatically, creates information repositories accessible rh rough 
any HTTP-compliant browser. Users can browse or quciy a repository for 
items of interest. 

In thus at tide, we explain the InfoHarness approach to metadata extrac- 
tion and heterogeneous information management. Wc also describe Visu- 
alHarness, which extends the basic system to accommodate visual data. 
Finally, wc describe how to use InfoHarness tools and hooks to create other 
Web-based, information- in tensive applications. 
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METADATA-CENTERED 
APPROACH 

Other researchers have investigated the use of 
metadata to support runtime access to original 
information 1 and data warehousing and mining 
for automatic metadata extraction. 2 In building 
Info Harness, we refined and synthesized these 
ideas to provide advanced search and browsing 
Capabilities without imposing constraints on infor- 
mation suppliers or creators. Moreover users can 
logically model the information space to best fit 
dteir needs. 

Our goals with TnfoHarness were to 

m provide integrated access to a networked het- 
erogeneous information source without forc- 
ing information relocation or reformatting, 

■ create and manage metadata for easy retrieval 
and decision Support, 

■ categorize related information items into col- 
lections to provide a logical information 
organization, 

m allow scalable searches, 

B create and manage relationships among groups 
of information items without affecting the 
information artifact contents, 

■ perform programmatic actions on retrieved 
information, and 

■ make information access and dissemination 
easy, low-cost, and ubiquitous. 

We translated these high-level requirements into a 
scalable, extensible archi lecture. 

Metadata Classification 

Wc classify pieces of metadata by bow successfully 
they capture the data and information content of 
documents from various media types. Metadata Can 
be content-dependent, content-descriptive (a spe- 
cial case of content dependence), or content-inde- 
pendent. In modeling application domain-specif* 
ic information, it is crucial to caprure the semantic 
content at a level of abstraction similar to that a 
human would employ. 

Content-dependent metadata depend only on 
the original data's conrcnt. It is easy to process 
text to identify such metadata, usually represent- 
ed as keywords, but for visual information it is 
very hard to extract. When this kind of metadata 
is not extracted automatically from the content 
itself, we call it content-descriptive. Content* 
descriptive metadata come from an analysis of rhe 
content. When this is not possible, chcy are 
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derived intellectually. An example would be to 
identify that a flower in an image has a "sweet 
and rosy" fragrance. Examples of content-descrip- 
tive metadata are the document vectors in Latent 
Semantic Indexing 5 and the complete inverted 
Wide Area Information Services index' 1 — these 
list the frequency and position of text units in a 
document. 

Content-descriptive metadata can be domain- 
dependent or -independent. Identifying chat a par- 
ticular shape of an ob ject in an image is that of a 
particular type of an airplane, for example, is 
domain-dependent metadata. A description of the 
structure of a multimedia document is domain- 
independent metadata. 



Inf© Harness is 0 mef&d&tcs 
management system with a 
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Creating content-independent metadata is like 
attaching a tag to the data regardless of its contents. 
Examples arc a document's creation date and 
location. 

InfoHarness Metadata Infrastructure 

InfoHarncss is basically a metadata management 
system with a generic metadata storage system as 
its nictabase. The system extracts metadata from 
information artifacts and then creates metadata 
objects that represent the original artifacts with 
related attributes. 

Hie system uses these metadata attributes to per- 
form various tasks. For example, InroHarness took 
could use a keyword list associated with an arti- 
fact — a content-dependent attribute — to build a 
keyword index. In addition to full-text and keyword 
searches, InfoHarncss lets users perform searches on 
metadata attributes. These prove especially useful 
when the user has some knowledge about the infor- 
mation artifacts' metadata semantics. 

The system's metabase consists of InfoHarncss 
objects that encapsulate the physical data repre- 
sented in the metabase. Each IHO can have any 
number of metadata attributes. The InfqHaxness 
server uses the metabase at runtime to build the 
user interface, browsing structure, and so forth. 
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Metadata Extraction and 
Management 

Info Harness can p reprocess information artifacts 
co generate metadata, or it can extract the metada- 
ta at runtime. ITic system's extractors automatical- 
ly gene rate metadata based on the media type. For 
example, a text extractor filters out relevant words 
and indexes them. Metadata extractors for dates 
and subjects generated from mail messages return 
lines starting with the keywords <&ftrand subject C 
or C++ extractors recogni'ie logical constructs such 
as functions, classes, and subclasses. 

Extracting domain- and content-dependent 
metadata from images would involve anticipating 
the range of user queries and is not feasible except 
for cases when a domain is highly controlled and 
well understood. Instead, one could extract 
domain-independent but content-dependent meta- 
data such as color and shape during preprocessing 
and other information such as patterns and outlines 
at access time," 5 Contcnt*indcpcndcnt metadata 
such as size and location arc of course available dur- 
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ing preprocessing. Metadata such as container hier- 
archies for multimedia can either be extracted or 
explicitly Supplied by the metabase designer. We are 
currendy wotking on automatic or semiautomatic 
generation of do main -dependent metadata, which 
helps in associating semantics with the contents. 

Extracting any type of metadata is dependent on 
the range of user queries. For example, a query 
might use the size of the data as a retrieval criteri- 
on when transport costs arc important. Also, meta- 
data could control the presentation and dynamic 
composition of retrieved information. 

An LnfoHarness prototype used IUustra to inves- 
tigate how an objecwelational database could be 
more effective for querying the metadata itself and 
letting the user browse stored metadata prior to 
building queries. Systems set up this way also let 
users manipulate metadata in different media cypes 



intuitively. For example, we might use metadata to 
relate unstructured data such as images with their 
structured data representations. In such cases it 
would be advantageous to store the metadata as 
annotations to the original information. 

If InfoHarness stores the metadata this way, it 
can easily modify the metadata to reflect changes 
in the information contents or the content descrip- 
tion!; — location,, for example. Such systems can also 
encapsulate type-specific functions along with the 
metadata to allow associative searches for unstruc- 
tured information by using the metadata repre- 
senting its features. An example would be to asso- 
ciate an image-processing function for land cover 
identification with a certain type of satellite map. 
The content-descriptive metadata that the system 
stores— container hierarchies for a multimedia 
object, for example — determine how the user 
browses the information. 

Another issue in metadata storage Is the meta- 
data's location for remote queries. The system can 
store metadata locally or at remote sites, or it can 
prefetch the metadata for a query and then use it 
to intelligently analyze the query. For example, if 
the system determines from the metadata that the 
querying site cannot handle the size of the results, it 
can take appropriate action rather than retrieving 
the information and then failing. 

Logical Structuring and Browsing 

The InfoHarness metabase has an object layer over 
it, which it manages as a relational database. This 
metabase represents Teal- wo rid information arti- 
facts as encapsulated metadata IHOs, and it 
includes constructs thai allow arbitrary typed rela- 
tionships between IHOs. The IHOs can be simple, 
directly encapsulating real-world information arti- 
facts, or abstract, representing ex is ting Info Harness 
Structures. One such abstract IHO type is collec- 
tion, a logical grouping that the InfoHarness server 
interprets to provide a browsing and searching 
structure. That is, the browsing structure with 
which the user navigates die InfoHarness reposito- 
ry is equivalent co the collection hierarchy. The 
metabase is available to multiple InfoHarness 
servers and remains persistent between their mul- 
tiple instances- 

Because die metabase uses a traditional database 
management system,, it can handle concurrent 
updates and can be dynamically extended and mod- 
ified while the InfoHarness servers axe cunning. This 
dynamic persistent metabase has a clean interface to 
the server on one end and the InfoHarness admin- 
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Lsrracive cools on the other. However, as extractors 
return objects and InfoHarncss stores diem in a rela- 
tional database, the system faces the limitations and 
complexities of any object-relational mapping. 

LOGICAL METADATA-BASED 
INFORMATION SPACE 
STRUCTURING 

Artifacts become available through InfoHarncss 
after die system registers them. During registration, 
the .system extracts metadata from the artifact and 
creates an IHO encapsulating various pieces of 
metadata about it. The IHO serves as a handle for 
the actual artifact. Each IHO contains several 
attrib uted m etadata — s o me m an dato ry a nd m ai n - 
tamed by the system, such as document type. 
objeel ID, and artifact location. There is no fixed 
schema of attribute names in InroHarness, so appli- 
cation builders can create new attributes as need- 
ed without modifying the code. 

Each IHO has a particular object type attribute 
indicating the representation or format of the bytes 
making up the artifacts information and the seman- 
tics interpreting this representation. For example, a 
document type called BeHcore-Framervf alter- Doc- 
ument might have a FrameMaker binary storage 
format and semantics designated by one of the Bell- 
core standard Frame templates. InfoHarncss can 
handle any document type that represents a class of 
documents. The document types specification 
dcclates important information such as how to 
extract metadata and text from an artifact and what 
legal actions a user can perform on an IHO. 

Each document type has an extractor that pulLs 
out a certain fixed scr of metadata attributes. For 
example, the HTML extractor pulls out a docu- 
ment tide, while a Frame extractor pulls out addi- 
tional values based on FrameMaker variables. The 
extractors can be any script or executable on the 
system, but they must be wrapped to conform to 
the standard administrative interface. For example, 
a thin Perl script could wrap an existing trofT extrac- 
tor, reformatting the extractor's output into the 
TnfoHarness message exchange format. 

We mentioned earlier that there is no fixed 
schema over an entire InfoHarncss repository. In 
addition, there is no fixed schema for a single docu- 
ment type. Differ cju documents of the same type 
might have different sets of attributes, none of which 
are ever enumerated in any document type schema. 
This flexibility lets an application builder create 
applications that use new metadata attributes with- 
out havfng to modify- system schemas* 



When InfoHarncss registers an artifact, it stores 
the artifacts IHO in a repository. Most sites have a 
single repository holding all IHOs, but some cre- 
ate multiple repositories for large-scale partitions 
of their information space. Iwo groups sharing the 
same InfoHarncss software installation might create 
a repository for each project. Each installation has 
a single mctabase storing all metadata. 

InfoHarncss uses collections as modeling con- 
structs to organize the IHOs in a repository's in for- 
mation space. A collection represents a set of other 
related IHOs. For example, a user could create a 
collection containing all design documents for a 
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particular software package release. Collections can 
also contain other collections; this nesting forms a 
repository's collection graph. The collection model 
consists of sets of relationship structures imposed 
upon a repository rather than containment struc- 
tures in which to place and store objects, Collec- 
tions do not encapsulate information artifacts but 
participate in relationships with other IHOs or col- 
lections; this lets the user build arbitrary logical 
models. A collection-membership relationship 
exists between each collection object and member 
object. InfoHarncss understands this kind of reU- 
tionship and can interpret it during browsing and 
searching. The application-building tools could 
interpret these relationships in any context a 
designer desires. 

Users can set up annotation relationships among 
artifacts or relationships correlating all documents 
about a certain subject without explicitly including 
them in a collection. Each IHO can also participate 
in multiple relationships and be part of multiple col- 
lections. This lets users impose mure than one log- 
ical view on a given set of IHOs- For example, cer- 
tain users might want to browse a company 
personnel repository by departments and groups, 
whereas others might want to browse the same 
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Figure 1 . High-level InfoHarness architecture showing the three main system components. 



repository by subject expertise. We could impose 
both these views— and collection structures — on 
the same repository. 

An administrator can designate that a collection 
build a content index from the text of the collected 
documents. In this case, the extractors pull out the 
text body or a selected segment of the text from 
appropriate document types and pass them back to 
the administrative system. The system then uses the 
extracted text bodies to build the keyword index. 
Upon locating arty of these text bodies through a 
keyword, search, InfoHarness automatically refers the 
user to die [HO that encapsulates the information 
artifact associated with the text body; For such col- 
lections, the user can submit a content-based query, 
which uses the index to find documents that contain 
user-specified, keyword-based search Criteria. 

System Architecture 

Figure 1 depicts a client-server system in which the 
clients are HTTP-compliant Web browsers and the 
servers are InfoHarness servers providing access to doc- 
uments registered with a particular repository. When 
a user connects to an InfoHarness server* the system 
dynamically produces HTML pages that constitute 
the user interface. By activating links and using 



HTML forms, the user can navigate, search, and 
access the information maintained by that server. 

In response to user requests, the server can invoke 
multiple associated methods for each IHO. For any 
document type, users can define cypc-specific meth- 
ods for displaying the artifact associated wirh an 
IHO. The simplest mediod involves using an 
appropriate MIME type to display the original arti- 
fact on the client browser. In a more complex 
method, a user might click on an object, and Info- 
Harness could send the underlying artifact as an e- 
mail attachment. Other methods might translate 
the artifacts into HTML to allow the clients to dis- 
play them inline. Methods also let users perform 
tasks such as sending a document to the printer or 
sending a facsimile of the document to another user. 

A single IHO can be associated with multiple 
methods; the user can choose among them at run- 
time. For example, to display IHOs encapsulating 
Unix manual (man) pages, the user could choose 
the Unix utility xman or have InfoHarness trans- 
lace the pages to HTML and display them within 
the Web client. 

InfoHarness also has an extensive set of admin- 
istrative tools for managing authorization and 
security. 
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Figure 2. The VisualHarness architecture showing the use of VIR as a black box. 



VISUALHARNESS: VISUAL 
INFORMATION MANAGEMENT 

The VisualHarness system extends the basic Tnfo- 
Hamcss to deal with visual data, giving users the 
abilicy to search and access distributed repositories 
of text, images, and other types of data, including 
structured databases. In addition to keyword and 
attribute-based queries, it also supports content- 
based queries over images, involving color, texture, 
composition, and structure. A VisualHarness 
metabasc can consist of indices (for example, a full- 
text index for textual data and a feature-based index 
for image data) and attribute- value pairs. The 
attribute-value pairs support attribute- and con- 
tent-based access of visual data using a novel 
approach that converts feature vectors into struc- 
tured metadata. Like InfoHarness, VisualHarness 
has an open and extensible architecture that pro- 
vides hooks for using various third-party indexing 
engines for textual data and visual informarion 
retrieval (VIR) engines, such as VirageV 5 

Figure 2 shows a high-level view oi the Visual- 
Harness architecture. The InfoHarness server 
accepts a ttser query as a diem request from a brows- 
er, and rhe query engine module of the query-pro- 
cessing unit (QPU) creates subrequests for the rele- 



vant search components. The search components 
use metadata (p recomputed and stored in the 
metabase or Computed at runtime) to determine ref- 
erences to the relevant data and then provide them 
to the QPUs result composition module. The QPU 
normalizes, rcscalcs, and formats the result, which 
the InfoHarness server then displays to the user. 
When the user selects one or more data objects for 
display, the server directly accesses the appropriate 
repositories to retrieve the data. 

The query-processing subsystem uses weighting 
strategies for a scalable approach. That is, a user can 
assign different weights to different kinds of simi- 
larity. The system then restricts database informa- 
tion retrieval according to the assigned weights. 
Suppose the VIR engine supports three proper- 
ties — P 1( T* 2 , and P 3 . The user can assign each of 
these properties different weights — ij, ij* and i 3 — 
so that the VIR bases its retrieval on 

ijl'l + i^Pj + I3P3, where 0.0 £ 1^12,13 :& 1-0. 

The VIR normalizes and scales the resulting values 
to rank each of the objects retrieved. This access 
method applies if the system can access and under- 
stand feature vectors. Because VisualHarness has 
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Figure 3. Comprehensive search screen in the VtsualHarrtess system. 



neither access to the actual feature vectors of an 
image object nor a way of in expecting them, it uses 
the VTR engine as a black box. 

The Black-Box Approach for 
Content-Dependent Metadata 

Feature vectors from an image refer to the features 
extracted from different topological spaces. To rank 
the similarity of objects to a given query object, a 
system requires distances he t ween the objects and 
the input query object. The black-box approach 
compares objects based on their deferences with a 
reference image. 

If R is a reference image and Q\, Oj, O tt are 
the objects in the database, the feature distance (the 
distance between any two objects Oj and 0 2 in the 
feature space) equals the absolute value (Euclidean 
distance, denoted by of the difference between 
each object compared with the reference image for 
a particular property. Hut is, 

D(0!,0 2 ) = abs[D(0 |f R) -D(0 2i R)J. 

Feature vectors of the object sequence in the data- 
base based on different properties of an image ate 
mapped to a point in the feature space; a query 
with tolerance e becomes a sphere of radius e. 

The black-box approach allows high scalability 
because it docs not limit information retrieval to a 
particular VIR engine and its corresponding image 
database. Runtime computation is not expensive, 
because the system p recomputes the distance 
between each object and the reference image for 
each of its properties and stores these values in a 
roetahase. Runtime computation basically involves 



retrieving the appropriate results from the 
database by converting the user query 
image, Q, into a database query D(Q,R) — 
the distance between image Q and the ref- 
erence image. 

Without this approach, during runtime 
the system would have to sequentially com- 
pute the d is ranee between the query image 
and each image object in the databases. The 
black-box approach also lets us use different 
weighting strategies to combine the dis- 
tances obtained in comparing each object 
with the reference image in that topological 
space. Because we are using normalized dis- 
tances, we can also combine features com- 
puted using different engines. 

Our initial black- box straregy was go use 
a null image as the reference. We chose an 
entirely black or an entirely white image, hypoth- 
esizing thai such an image has no specific features 
and hence no properties of its own, and we 
obtained quite decent results. However, we con- 
tinued to seek a reference image that would yield 
results more accurate than those obtained using the 
VTR engine directly. In our second strategy the ref- 
erence image is the "ccntroid" of the feature space, 
Ideally, this object should be equidistant from alt 
the other objects. Because such an object would be 
difficult to construct* we chose an existing object 
close to the ideal location in the feature space. 

We also investigated improving results by 
scmanticaily correlating various objects into seman- 
tic groups. Members of such a group would have 
some binding feature, and objects coM belong to 
multiple semantic groups. That is, wc could 
"thread 11 objects based on some predefined seman- 
tics. By scmanticaily correlating the objects, we 
make an effort toward better understanding the 
intent of the query. Wc investigated bodi content 
semantics and conrext semantics. Grouping based 
on concent semantics relies purely on sratistical 
principles and can be mathematically formulated, 
whereas context-based grouping might be auto- 
mated, manual, or knowledge-driven. For further 
discussion of the black-box approach and various 
strategics for selecting reference images, including 
quantitative evaluations, sec our other work.* 

Figure 3 shows an example of the comprehen- 
sive search screen. A user could adopt any of the 
three search strategies (keyword-, attribute-, con- 
tent-based) or combine them using relative weights. 
For images, within the content-based search, the 
user could add color, structure, texture, and corn- 
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position ro the search criteria. Iterative refinements 
are also possible. 

APPLICATION BUILDING 

The InfoHarncss platform supports a wide range 
of custom h at ions, such as additional document 
formats and screen layouts. An organization could 
customize the user interface headers and footers, or 
it could build a periodically running script char 
looks for expired documents and notifies the 
authors or a responsible administrator by e-mail, 
In addition, InfoHarncss hooks let users create 
entirely new applications that work with a docu- 
ment control system, use a new indexing technol- 
ogy, or integrate with a billing and ordering system. 

The major steps in using InfoHarness to build 
an application arc 

* Model and create Info Harness collections and 
relationships. 

■ Decide which documents arc pan of which col- 
lections. 

■ Set application parameters. 

■ Design the process by which the application 
keeps information up-to-date. 

V Design scteen templates and widgets, imple- 
menting extractors if necessary. 

This job requires a thorough understanding of 
information content and flow within the applica- 
tions scope. The application builder must under- 
stand how the end user wants to find and access 
mrormatf on, keeping in mind special requirements 
such as authorization and security. 

For those who wish to extend the predefined sys- 
tem's functionality, Info Harness gives system inte- 
grators an API to add programs or scripts. Sysiem 
integrators can, for example, enhance the applica- 
tion with additional document types or actions on 
document types, or add support tor a new indexing 
package. Because document type methods can be 
any arbitrary script or wrapper, many integration 
scenarios are possible. Specific extractors and run- 
time methods let the application builder integrate 
legacy dura, systems, and applications. These lega- 
cy systems could be wrapped and encapsulated as 
IHOs to suit die particular Web-based application. 

Mow lets take a step back to examine the activ- 
ities that are likely to be involved in building an 
application. 

Information Inventory and Modeling 

Taking inventory of the most valuable information 
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can be as simple as consulting project documenta- 
tion control or as difficult as creating a special-pur- 
pose committee for the task. Several items are use- 
ful at this stage: 

■ a draft inventory of all frequendy used resources, 

■ a scenario describing expected information con- 
sumers and usage patterns, 

■ a list of entry points into the organization for 
new information, and 

■ a few friendly users who agree to provide feed- 
back on the application. 

Information modeling involves defining the 
structure through which users will find resources, 
This organization is important in the applica- 
tion's perceptual ease of use. We suggest these 
guidelines: 

m Depth verms xvidvh. A primary decision will be 
how much information to put at any given level 
and how many levels to create. Deep applica- 
tions need many specific collections, while 
broad applications need fewer general collec- 
tions. The more clicks it takes for users to find 
whal they arc looking for, the more likely it is 
that they will end up in the wrong place. 

B Customer focus. Information- modeling interests 
will conflict, but the most frequent and impor- 
tant users should expect the smoothest ride. If 
technical people are the intended primary users, 
then technical collections should be at the top 
icveL If managers or customers arc the prima- 
ry users, technical details might be several levels 
down, available when necessary. 

■ Frequency of use. Make a collection of hot items 
chat people refer co most frequently. This could 
contain, for example, project status reports, 
templates, or subject matter expert lists. 

B Indexes at appropriate levels. Global indexes arc 
useful for shot-in- the-dark searches, but tbey 
may not be focused enough for experts hunt- 
ing for a specific detail- Consider, for example, 
a software- engineering repository that contains 
collections for designs, code segments, require- 
ments* test cases > and so on. A developer search- 
ing for reusable components will be better off 
searching an index of the code collection radicr 
than rhe entire reposirory. 

■ LeUrning by example. Survey related informa- 
tion systems and study their organisation. 
Many information applications on the Internet 
illustrate good ways to model data. 
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Once you have drafted the collection structure, you 
might prototype a subset of the application and 
have friendly users provide feedback. 

Resource Translation 

Although InfoHarness includes a wide variety of 
rcady-to-use extractors* these cannot cover every 
conceivable application need. Writing new extrac- 
tors can be trivial, provided you have the necessary 
technology to parse the given document type. 
Then, you simply plug in the new extractor. 
Recompiling the InfoHarness server is not neces- 
sary, although you may have to recompile the 
extractor if it is not written in a scripting language. 

The logical application model need not have any 
relation to the physical distribution of the original 
documents. Once you set up the logical model* 
InfoHarness completely shields the user from the 
underlying physical structure. In rare cases, the 
application builder might move data to a more cen- 
tral location or convert it to a single format. This 
makes building an application simpler, but Info- 
Harness can bring together widely distributed 
information as well. 

Interface Design 

Users usually require help or additional details. 
Place necessary guidelines* help screens, hypertext 
links* method launch links, and contact informa- 
tion in strategic places. For example, the server uses 
screen templates or widgets to dynamically gener- 
ate the search screens; the application builder could 
customize these templates to the users' needs. 

Initial Repository Creation 

Bring the application resources (or the metadata) 
into a. single place, possibly with searchable index- 
es for some or all sets of data. This involves using a 
set of administrative tools to build up a sot of 
objects and relationships in the merabase. Use the 
InfoHarness tool set to wrire higher level scripts 
that define the repository structure and determine 
how collections arc built. 

Repository Maintenance Mechanism 

When an application demands timely update of 
resources, establish automated procedures to find 
new resources, translate them if desired, and add 
them to the repository in the correct groupings. 
Maintenance scripts exist, but these do not auto- 
matically update resources or find new resources. 
You will have to write your own daemon (or agent) 
for that. 
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System Integration 

If you arc integrating one or more cxis ting systems 
or interfaces, you must supply specific wrappers. 
InfoHarness has a welUdeflncd flexible interface, 
which can bring di fie rent systems together. Con- 
sider a personnel application that has its original 
artifacts in a legacy system. The system integrator 
has access to scripts that can run as clients on the 
Unix side and that execute queries on the main- 
frame back end. The extractors could be wrappers 
for these Unix scripts. 

Suppose an IHO corresponds to an employee: 
Given an employee identification number, the 
extractor wrapper could invoke the existing scripts 
and return metadata from the legacy system that, 
after reformatting, would be encapsulated and 
inserted into the metabasc. At runtime, a similar 
wrapper would invoke the existing scripts to dis- 
play employee information to the user. Employee? 
could be included in multiple collections based on 
group> department, or other factors. 

Client Configuration 

Setting up the Web browsers with die proper view^ 
ing utilities can be a major consideration, depend- 
ing on which data formats the application will use. 
Unix machines cannot view PC-created documents 
without considerable translation or sophisticated 
tools. IrtfbHarness cannot directly aid in this phase, 
but its support of extensible methods can make the 
task simpler. 

The InfoHarness commercial version (see the 
sidebar, "Related Work") has a PC patch called PC- 
AdaptX, which provides extractors for Microsoft 
Office and other documents. However PC-AdaptX 
uses die InfoHarness utilities provided on Unix, so 
you should have a PC network fJe system (PC-NFS) 
set up to cross-mount your PC file system on your 
Unix platform. 

ONGOING WORK 

This article docs not cover all of our work on Info- 
Harness. One InfoHarness extension provides 
access to relational databases. This lets users browse 
and have keyword- and attribute- based access to 
collections of objects defined over relational rabies 
from multiple DBMSs. The extended system also 
supports audio, video, 3D, and other spatial data. 
Other recent work on InfoHarness has extended 
the query subsystem co let users combine attribute, 
keyword, and image feature searches over InfoHar- 
ness repositories. We also researched the issue of 
scalability by allowing access to and combining 
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results from multiple data partitions, each with its 
own independent and possibly different indexcr. 
We have also made infras true cure layer extensions 
to allow the use of CORBA for distributing the 
IrifoHarncss repositories themselves. 

Taking the second-generation lnfoHarness con- 
cepts' a step further leads us co the InfoQuiit sys- 
tem,* a third generation of information interoper- 



ability and integration that we are currently at work 
on. InfoQuiit s purpose is to provide Intelligent 
analysis, mining, fusion, and dissemination of het- 
erogeneous media data. Its architecture has three 
layers — data, metadata, and user/domain models 
or ontologies. With extensive support for metada- 
ta and user models, InfoQuiit will define context, 
support user profiles and models, use mcdia-inde- 
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pendent correlation to represent media-indepen- 
dent semantic relationships, 9 and eventually sup- 
port multiple domain-specific ontologies, 10 Infb- 
Quilt will allow traditional keyword-based queries 
as wcl! as high-level information requests involving 
attribute-based, iconic, mixed-media, and concept- 
based information requests. This will involve dis- 
tributed management and correlation of heteroge- 
neous media stored at different locations. The 
system will also support information analysis and 
fusion at a higher semantic level with the help of 
human-directed browsing, searching, accessing, and 
correlation of heterogeneous media data. ■ 
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